Independent Proprietorship and Competition in Distributed Web Search

The predominant Web search model attempts to use multiple computers under centralised management to act as one search engine for the entire Web. As the quantity of online information increases, systems based on this model become prohibitively expensive for all but the largest organisations. We advocate the use of distributed search architectures where multiple independently owned and managed search engines act as one search system. This approach has significant advantages including low market entry cost for individual search providers and the potential to stimulate the provision of high-quality services through competition. The low entry cost allows small organisations and even individual users to influence service features and quality by establishing specialised search services. However, independent proprietorship also greatly complicates the search system design. The potential for competition between engines requires new approaches to effective engine management. Many new issues arise such as deciding what information an engine will index. In this paper, we analyse the sources of complexity in distributed Web search architectures with independent proprietorship and competition between engines. We outline possible ways to cope with this complexity using techniques from the field of computational economics such